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EXECUTIVE  SUMMARY 


This  research  was  conducted  to  understand  whether  complex  information  processing 
affects  team  performance.  Specifically,  information  display  monitoring  was  analyzed. 

The  study  used  both  subjective  and  objective  measures.  The  subjective  measures  were 
used  to  assess  team  effect  in  terms  of  team  situation  awareness  (SA)  and  team 
information  sharing.  The  objective  measures — signal  sensitivity  (d’),  time  and  accuracy 
were  used  to  analyze  effects  of  task  performance,  specifically,  signal  monitoring.  There 
was  no  difference  between  the  teams  used  in  the  study  with  respect  to  how  they  reported 
seeing  the  same  thing  (team  SA)  and  exchanging  information  (team  informity).  For  the 
team  with  3  members,  the  distributions  of  the  mean  percentages  of  team  S  A  tend  to 
follow  exponentially  decreasing  function  (negative  slopes)  as  the  number  of  signal 
presented  increases.  The  team  with  5  members  reported  more  team  SA.  It  is  not  clear 
from  these  results  whether  teams  with  more  members  tend  to  identify  more  signals  than 
teams  with  less  number  of  members  (G3).  It  is  possible  that  signal  complexity  and  signal 
timing  control  may  interact  to  affect  signal  monitoring  performance — which  in  turn 
affects  the  way  people  report  what  they  see  under  stress.  This  was  indicated  by  the 
interactions  between  signal  complexity  and  signal  timing  control.  The  distributions  of 
mean  team  informity  show  increasing  positive  slopes  under  different  signal  timing 
controls.  Team  G3  shows  the  least  report  on  information  exchange  among  members  at 
signal  control  time  of  0-5  sec,  but  scored  high  when  the  signal  timing  was  random.  Team 
G5  exhibited  more  information  sharing  among  its  members  at  signal  time  control  of  10- 
15  sec.  Signal  detection  sensitivity  was  calculated  across  all  experimental  blocks.  In  team 
G3,  single  or  multiple  signals  presented  randomly  (random  time  control)  tend  to  reduce 
the  operator  detection  sensitivity.  It  is  not  clear  why  three  signals  with  timing  control 
levels  of  5-10  sec  and  10-15  sec  induce  poor  sensitivity.  The  best  d’  for  G3  team  occurs 
when  monitoring  4  signals  at  10-15  sec  time  control  (d’  =  5.15)  and  also  when  2  signals 
are  presented  at  the  same  time  control.  With  G5  team,  a  higher  d’  occurs  at  a  timing 
control  of  5-10sec  for  a  single  signal  monitoring  task  (d’  =  2.96);  the  worst  d’  occurs 
when  2  signals  are  monitored  under  random  time  control.  In  this  study,  interactions  for 
mean  detection  accuracy  are  more  pronounced  at  signal  timing  control  of  10-15  sec.  G5 
team  shows  two  possible  interactions  at  random  signal  timing  for  signal  complexities  1 
and  2,  respectively.  In  terms  of  detection  (response)  time,  both  teams  tend  to  perform 
almost  without  statistically  mean  difference. 
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CHAPTER  1 
STUDY  RATIONALE 


The  primary  cause  in  many  accidents  is  the  Captain's  failure  to  control,  and  the  Co¬ 
pilot's  failure  to  monitor.  Crew  members  are  the  final  opportunity  to  stop  errors,  but  the 
Crew  are  also  human  ( http://www.wcupa.edu/  ACADEMICS/sch  cas.Dsv/Career  Paths/Aviation/subfield2.htm) 


1.1  Prolegomena 

To  better  gain  the  thematic  knowledge  of  this  report,  let  us  review  some  incidents  that 
can  be  attributed  to  computer  display  and  visualization  information  processing  in  teams 
or  at  individual  levels. 

(a)  On  April  12,  1999,  BBC  World  News  reported  that  a  train  was  destroyed  as  a 
result  of  a  timing  error.  Russia  and  China  lodged  protests  after  Nato's  raid  on  a 
railway  bridge  hit  a  passenger  train,  killing  10  and  leaving  16  injured.  The 
Belgrade-Salonika  train  had  been  crossing  the  bridge  near  Leskovac,  southern 
Serbia  as  the  air-launched  missile  released  several  miles  away  reached  its  target. 
Showing  missile-cone  video  footage  of  the  attack,  Nato's  commander  of 
European  forces  Gen  Wesley  Clark  said:  "You  can  see  if  you  were  focusing 
right  on  your  job  as  a  pilot  how  suddenly  that  train  appeared.  It  was  really 
unfortunate. "  In  monitoring  large-scale  information  systems,  these  “pop  ups” 
are  considered  surprises  or  threat  incidents  to  be  watched  for. 

(b)  On  April  14,  1991,  many  refugees  were  bombed  as  a  result  of  mistaken  identity. 
After  days  of  speculation,  Nato  leaders  said  the  alliance  was  responsible  for  two 
attacks  on  Kosovo  convoys,  both  of  which  might  have  included  refugees  fleeing 
western  Kosovo.  The  alliance  said  that  its  pilots  had  been  hunting  Yugoslav  units 
burning  out  villages  in  western  Kosovo  but  Serb  media  said  the  attack  left  70 
dead.  The  picture  was  further  muddied  after  it  was  revealed  that  a  recorded 
debriefing  of  a  Nato  pilot,  released  to  the  media,  was  not  the  airman  responsible 
for  the  strikes.  "This  is  a  very  complicated  scenario  and  we  will  never  be  able  to 
establish  all  the  exact  details, "  said  US  General  Daniel  Leaf. 

(c)  At  the  end  of  the  Persian  Gulf  War,  two  American  A- 10  Warthog  pilots  fired  on 
what  they  thought  was  an  Iraqi  armored  column  that  turned  out  to  be  a  group  of 
British  fusiliers  and  nine  were  killed.  A  senior  military  spokesman  noted  that 
perhaps  the  “telemetry  information”  may  have  been  confusing. 

(d)  On  April  1 8  incident  at  Tamak  Farms,  near  Kandahar,  two  Air  Force  pilots  saw 
muzzle  flashes  at  an  established  firing  range  20,000  feet  below  and  mistook  them 
for  antiaircraft  fire  aimed  at  their  F-16's.  One  pilot  dropped  a  500-pound  bomb, 
killing  four  Canadian  soldiers  and  wounding  eight. 
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The  above  scenarios  capture  some  stories  relevant  to  information  processing — 
either  in  teams  or  individuals.  In  all  cases,  there  are  some  elements  attributable  to 
complexity — such  as  “complicated  scenario”,  “unexpected  train  appearance”, 

“confusing”,  and  “mistaken  identity.” 

Decision  making  in  time-critical,  dynamic,  and  complex  environments  involves 
many  complex  processes  that  include,  for  example,  the  use  of  information  display  and 
visualization  and  teams  of  decision  makers  such  as  the  military  pilot  and  co-pilots.  The 
information  streams  are  a  mixture  of  human  and  technology  driven  sensors.  Singh,  Tu, 
Allanach,  Areta,  Willett,  and  Pattipati  (2006)  observed  note  that  “Not  only  are  the 
sources  of  information  diverse,  distributed,  and  possibly  conflicting,  but  also  the 

acquired  information  is  very  likely  noisy,  dynamic,  incomplete,  and  uncertain . The  key 

issues  involve  not  only  identifying  valuable  information  in  a  timely  fashion,  sharing  this 
information  across  agencies  in  an  efficient  manner,  but  also  integrating  volumes  of 
disparate  information  to  support  strategic  decision  making  (pp.9) 

Decision  making  is  the  cognitive  process  of  deciding  upon  a  particular  course  of 
action  based  on  available  data.  Cognitive  Science  has  been  able  to  create  a  substantial 
body  of  data  on  the  decision  process  (Brehmer,  1984;  Ho,  1980).  Today,  most  of  the 
decision  making  processes  and  the  decision  makers  are  supported  in  some  way,  by  the 
use  of  computer  information  display  and  visualization  models.  This  requires  cognitive 
and  neurophysical  resources  of  the  decision  maker. 

It  is  not  clear  how  information  display  for  teams  involved  in  dynamic  and 
collaborative  decision  making  improves  performance  and  mitigates  occurrences  of 
mishaps.  Instances  of  information  visualization  currently  range  over  many  domains  of 
human  knowledge.  For  example,  military  and  civilian  command  and  control  (C2)  centers 
depend  on  information  visualization  to  monitor,  diagnose,  and  take  actions  during 
emergency  conditions.  In  fact,  in  most  cases,  the  information  to  be  visualized  is  not 
static — as  it  changes  according  to  real-time  environment  dynamics.  More  so,  the 
displayed  information  is  used  by  an  individual  or  a  team  of  decision  makers  for  enacting 
the  necessary  system-level  actions.  Whether  the  information  is  presented  for  an 
individual  or  a  team  of  decision  makers,  the  density  and  volume  of  information  presented 
to  the  human  can  be  so  overwhelming  that  it  generates  stressful  conditions  that  may 
likely  result  in  overburdened  mental  workload  (Wickens,  &  Kessel,  1981),  resulting  in 
errors  and  costly  accidents.  The  sample  scenarios  above  illustrate  these  facts. 

Having  the  right  information  at  the  right  time  is  certainly  helpful — often  vital — 
for  any  decision  making  process.  However,  we  recognize  that  too  much  information,  or 
information  that  is  presented  in  a  way  that  overwhelms  the  human  sensory  or  cognitive 
system  can  also  fail  to  inform,  impairing  a  decision  making  process. 


1.2  Project  Scope 

The  effect  of  complexity  in  information  processing  is  addressed  in  this  study 
because  it  provides  a  compelling  view  of  the  existing  information-centric  work  domains. 
The  scope  of  this  project  is  limited  to  evaluating  the  performance  of  a  team  of  decision 
makers  (DMs)  processing  complex  information  during  a  crisis  situation.  The  complexity 
is  introduced  by  uncertainty  of  where  the  next  crisis  will  occur,  when  it  will  occur,  and 
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modalities  of  information  communication  between  team  members.  Team  characteristics 
such  as  differences  in  command  and  control  policies  and  procedures,  shared  mental 
models,  and  shared  situation  awareness  are  used  as  important  variables.  Specifically,  this 
research  will  emphasize  the  understanding  of  team  characteristics  as  predictors  of  team 
performance  in  a  complex  information  environment.  The  domain  of  signal  monitoring 
that  mimics  an  emergency  situation  is  used  to  validate  the  hypotheses. 


1.3  System  Monitoring  as  the  Major  Task 


Exhibit  1 :  Cockpit  Display 


Exhibit  2.  Nuclear  Power  Plant 
Display  (Yokogawa,  2006) 


Exhibit  3.  Air  Traffic  Controllers  Monitor  radar  Screens  to  Help  Aircraft  Reach  Their  Intended 
Destinations  Safely  ('http://www.bookrags.com/sciences/computerscience/aircraft-trafftc- 
management-csci-03  .html 


Complex  systems,  such  as  urban  traffic  flow,  air  traffic  control,  or  nuclear  power 
plants  are  usually  monitored  by  observing  computer  displayed  information  recorded  by 
sensors  placed  at  various  locations  in  the  system.  Data  from  these  systems  must  be 
monitored  and  crucial  life  and  death  decisions  made  by  personnel  within  a  short  period  of 
time.  Exhibits  1-3  show  examples  of  these  kinds  of  systems. _ 
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Complex  systems  require  increasingly  sophisticated  monitoring  systems.  Care 
must  be  taken  to  design  secure  systems  that  meet  requirements  and  are  perceived  as 
accurate  by  the  operators.  Far  too  many  accidents  have  taken  place  due  to  operators 
assuming  that  messages  were  false  positives  when,  in  fact,  the  alerts  were  accurate.  A 
complex  system  has  an  area  where  there  are  constant  false  alarms  coming  from  a 
monitoring  system  used  to  detect  a  security  breach  (Smith,  1995).  Research  in  vigilance 
and  automation  monitoring  provides  the  thematic  understanding  for  experiments  dealing 
with  complex  information  processing  of  this  report.  The  extent  to  which  teams  perform  in 
making  decisions  using  computer  displays  is  assumed  to  be  more  realistic  than  studying 
single  operator  performance  which  is  usually  the  norm  in  automation  monitoring  studies. 

1.4  Consequences  of  Poor  and  Adequate  Information  Processing 

Recent  data  has  shown  that  human  error  is  the  leading  cause  of  system 
unavailability  in  large-scale  commercial  data  systems.  The  accident  at  Three  Mile  Island 
Nuclear  plant  is  another  example  of  a  critically  complex  system  where  the  monitoring 
process  failed.  Bignell  and  Fortune  (1984)  give  a  critical  evaluation  of  the  whole  episode. 
Their  conclusions  show  that  the  failure  of  one  valve  and  the  way  the  information  was 
presented  to  the  operators  was  the  overwhelming  cause  of  the  disaster.  The  confusion  in 
the  data  and  the  amount  of  signals  to  be  monitored  by  the  operators  was  too  great  for 
effective  intervention. 

There  are  three  distinct  problems  associated  with  a  human  operator  or  a  team  of 
operators  in  a  complex  system  of  information  processing  and  decision  making.  These  are: 
visual  search  complexity,  judgment  errors,  and  poor  prediction.  When  we  are  looking  for 
objects  in  a  visual  scene,  we  often  have  to  perform  a  time-consuming  process  called 
visual  search,  where  we  choose  which  objects  to  attend  to  and  then  perhaps  perform 
additional  mental  processing  on  the  attended  items  to  find  what  we  are  looking  for. 
Human  Factors  scientists  have  studied  the  mental  processes  that  underlie  visual  search 
using  a  fairly  standard  paradigm,  called  a  visual  search  task  (Gai  &  Cury,  1976;  Molloy 
&  Parasuraman,  1996).  In  this  task,  a  number  of  items  are  presented  on  a  computer 
screen,  and  the  subject  searches  for  a  known  target  item.  Half  of  the  trials  contain  a  target 
item  mixed  in  with  distracter  items,  and  half  of  the  trials  contain  just  distracter  items.  The 
subject  in  the  experiment  indicates  on  each  trial  whether  the  target  is  present  or  not,  and 
the  target-present  and  target-absent  trials  are  presented  in  random  order.  Because  this  is 
usually  a  fairly  easy  task,  the  performance  measure  is  how  fast  the  subject  responds  with 
reaction  time  and  number  of  errors  committed  (Eriksen,  1952;  de  Boer,  1966). 
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CHAPTER  2 


COMPLEX  INFORMATION  PROCESSING 
ENVIRONMENT 


2.1  Prolegomenon 

C2  personnel  working  with  dynamic  and  uncertain  information  face  the  challenge 
of  continuing  operations  as  well  as  surviving  in  a  constantly  changing  environment. 
Complexity  is  an  unavoidable  part  of  modem  work  systems.  Complexity  theory  attempts 
to  tackle  complex,  interrelated  systems  by  looking  at  the  whole  of  the  system,  not  just  the 
component  parts.  Among  these  is  how  individuals  or  groups  process  complex  and 
evolving  information  in  a  network  of  system  of  systems  (Danielsson  &  Ohlsson,  1999) 

Complexity  is  a  relative  term.  Simon  (1979)  argues  that  information-processing 
psychology  has  been  pursued  at  two  levels:  at  the  level  of  the  “immediate  processor”  and 
at  the  level  of  “relatively  complex  human  performances”  (p.  xi).  Investigations  of 
memory-scanning  or  sentence-picture  verification  (see,  e.g.,  Clark  and  Chase,  1972) 
characterize  the  former  approach,  whereas  the  Simon  and  Kotovsky  (1963)  studies  of 
letter  series  tasks  are  typical  of  the  latter. 

Difficulties  relating  to  decision-making  in  complex  problem  solving  domains  can 
be  exacerbated  by  uncertainty  regarding  available  data  and  domain  knowledge.  A  range 
of  techniques  exists  to  support  decision-making  processes,  but  a  primary  factor  relates  to 
information  display. 

The  traditional  definition  of  a  complex  system  is  that  it  involves  a  large  number  of 
dynamically  interacting  elements,  leading  to  non-linear  behaviors  of  the  entire  system  or 
sub-systems  (Cowan,  Pines,  &  Meltzer,  1994).  In  today’s  information-centric  society, 
complexity  induces  a  network  of  interacting  information  in  a  dynamic  environment — 
with  evolving  characteristics  that  involve  uncertainties  and  equivocality. 


2.2  Dimensions  of  Information  Complexity 

The  state  of  the  world  can  be  predictable  or  unpredictable.  Predictable  states  mean 
that  the  behavior  of  the  system  is  deterministic — leading  us  to  have  some  estimate  of 
what  the  next  system  behavior  will  be.  Unpredictable  states  of  the  world  mean  that  we 
have  no  control  of  what  will  happen  next.  Dynamic  complexity  is  a  result  of  system 
whose  state  changes  are  either  predictable  or  unpredictable  with  respect  to  time  and 
changing  patterns  of  information.  Hill  and  Levenhagen  (1995)  note  that  unexpected  cues 
within  organizational  information  processing  can  lead  to  missed  opportunities  as  a  result 
of  disoriented  or  misaligned  mental  models — leading  to  loss  of  meaning  and  construction 
of  a  mapping  model  that  identifies  with  the  situation.  Table  1  illustrates  these  dimensions. 
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Table  1.  Simplified  Taxonomy  of  Complex  Information  System 
Information  Characteristics 


The  World 

Deterministic 

Random 

Predictable 

Simple 

Complex 

Unpredictable 

Complex 

Wicked 

Some  of  the  characteristics  of  information  complexity  that  may  affect  decision  making  by 
individuals  or  teams  of  decision  makers  are: 

1.  Information  is  often  highly  ambiguous — leading  to  possible  erroneous  and 
multiple  interpretations  or  understanding. 

2.  Information  may  have  multiple  meanings  in  a  context. 

3.  Information  patterns  are  dynamic  and  change  in  space  and  time. 

4.  Information  has  multi-traits,  multi-scales,  and  multi-attributes. 

5.  Information  states  are  usually  unstructured,  leading  to  disorganized  processing  by 
the  human — e.g.,  mapping  the  human  mental  model  to  the  information  space. 

6.  Information  presentation  by  technology  display  mode  may  create  many  forms  of 
uncertainty — leading  to  situations  where  cause  and  effect  linkages  are  not 
inherently  knowable;  e.g.,  effect  of  display  compatibility  (Wickens,  1992)  or 
mental  model  capture  (Johnson-Laird,  1998). 

2.3  Information  Processing  and  Task  Complexity 

The  complexity  of  search  and  monitoring  tasks  has  been  shown  to  be  an  important 
factor  in  searchers’  ability  to  find  relevant  information.  The  literature  suggests  many  task 
characteristics  related  to  complex  information  environment:  information  density,  search 
and  confirmation  strategies,  number  of  goals  and  conflicting  dependencies  among  them, 
uncertainties,  presentation  format  and  modalities,  number  of  inputs,  cognitive  and  skill 
requirements,  as  well  as  the  time-varying  conditions  of  task  performance  (Bystrom  and 
Jarvelin,  1995;  Campbell,  1988).  In  a  study  by  Ntuen  and  Rodgers  (2002),  information 
selection  was  found  to  be  a  limiting  factor  in  determining  predictive  accuracy  of  signals 
presented  randomly,  with  information  processing  being  a  limiting  factor  for  the  groups 
with  heavy  workload  (measured  by  information  density). 

2.5  Visualizing  Uncertain  and  Poorly  Constructed  Patterns  of  Information 

Visual  search  is  the  cognitive  process  of  finding  particular  data  within  a  visual 
field.  In  crisis  management,  visual  search  occurs  when  a  user  responds  to  a  situation  and 
requires  additional  information,  thus  requiring  the  user  to  search  the  crisis  management 
interface  for  the  relevant  information.  Gilmour  and  his  colleagues  at  Boeing  (Snyder, 
1970)  have  shown  that  for  nearly  all  air-to-ground  search  conditions,  the  observer  wastes 
more  than  40  percent  of  his  time  in  useless  search  activity  during  the  period  after  the 
target  has  become  available.  Several  studies  have  reported  that  search  times  are  related  to 
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the  density  or  complexity  of  the  background  (Bloomfield,  1984;  Eriksen,  1952;  Drury, 
Clement,  &  Clement,  1978). 

There  is  more  than  one  way  to  classify  how  uncertainty  can  be  visualized.  One  is 
by  how  uncertainty  itself  is  represented;  another  is  by  how  uncertainty  is  encoded  into 
visualization.  For  the  latter,  there  are  two  general  ways  of  combining  uncertainty  into 
visualization:  (a)  mapping  uncertainty  information  as  an  additional  piece  of  data,  and  (b) 
creating  new  visualization  primitives  and  abstractions  that  incorporate  uncertainty 
information. 

Information  uncertainty  is  ubiquitous  in  almost  every  decision-making  and 
problem  solving  situation.  It  is  a  major  source  and  characteristic  of  complexity. 
Information  uncertainty  can  be  induced  by  time,  location,  events,  situations,  and  tasks. 
For  example,  the  path  trajectory  of  a  car  driving  on  the  highway  can  induce  path 
uncertainty  with  respect  to  position  and  angle,  often  caused  by  speed,  vehicle  control, 
other  vehicles  and  external  information  on  the  highway. 
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CHAPTER  3 

COLLABORATIVE  TEAM  DECISION 


Modem  organizations,  including  the  military  establishments,  are  designed  around 
coalitions,  where  two  or  more  organizations  with  mutual  goals,  collaborate  to  pursue 
common  goals.  Typically,  collaboration  involves  a  situation  where  the  collaborating 
entities  share  the  same  information  space;  the  same  interests;  share  the  same  mutual  and 
consensus  strategies.  Depending  on  the  composition  of  the  collaborating  organizations, 
managing  the  evolving  collaborative  behaviors  can  be  complex. 

The  resurgence  of  research  effort  in  military  team  decision-making  has 
reverberated  across  military  and  civilian  organizations  as  emergency  and  preparedness 
models  are  sought  to  counter  the  dreadful  expectations  of  terrorism,  both  at  home  and  in 
U.S.  interests  abroad.  Emergency  management  offers  varied  challenges  to  the  many 
facets  of  information  processing — spanning  across  different  organizational  hierarchies  to 
first  responders  in  the  field.  For  example,  complex  and  interrelated  plans  from  different 
command  and  control  (C2)  agents  must  be  developed  and  executed  in  real-time.  Limited 
resources  must  be  carefully  managed  and  coordinated,  and  time-critical,  high-stake 
decisions  must  be  made.  Generally,  the  use  of  information  display  that  captures  the 
realism  of  ecological  niches  through  various  sensor  mechanisms  is  the  key  to  detecting 
potential  system  anomalies,  such  as  terrorist  threats,  component  failures,  and  other 
unwanted  events  in  a  system  operation.  Detecting  these  anomalies  through  predictive 
simulated  scenarios  will  help  decision  makers  to  prepare  for  “worst  case”  situations  in 
real  world  situations — an  example  is  the  recent  catastrophe  from  hurricane  Katrina. 

In  many  work  domains,  such  as  aviation,  pilots  keep  track  of  a  dizzying  amount 
of  visual  information  while  operating  commercial  and  military  aircraft.  Air  traffic 
controllers  on  the  ground  must  remember  the  contents  of  their  radar  displays  while 
executing  multiple  peripheral  tasks.  Similarly,  in  surface  transportation,  automobile 
drivers  must  remember  the  positions  of  other  cars  on  the  road  while  planning  their  route 
and  attending  to  road  signs,  traffic  lights,  and  dashboard  instruments;  the  subway  train 
operators  are  equally  tasked  to  monitor  many  levels  of  sensor  information  that  track 
inbound  and  outbound  buses.  By  monitoring  computer  displays,  most  tasks  involve  team 
work:  (1)  the  pilot  and  co-pilot  team  members;  (2)  two  or  more  people  monitoring  urban 
traffic;  (3)  a  team  of  physicians  viewing  the  same  display  while  making  surgery 
decisions. 

A  significant  body  of  literature  exists  on  team  decision  making  and  collaboration. 
Horvitz  (1999)  characterizes  the  types  of  collaboration  as  follows: 

•  Mixed  initiative.  The  parties  to  collaboration  have  a  mixed  initiative  relationship.  Any 
of  the  collaborating  parties  can  propose  information,  interpretations  and  solutions  to 
the  problem  being  addressed. 

•  Shared  purpose.  The  collaborating  parties  have  a  shared  sense  of  purpose.  There  may 
be  other  goals  of  the  parties  that  are  not  shared,  or  even  openly  contested,  but  the 
parties  agree  on  the  purpose  of  their  collaboration. 

•  Shared  situation.  The  collaborating  parties  must  interpret  the  conditions  of  the 
environment  in  the  same  way.  If  the  assessment  of  the  situation  is  not  the  same  at  the 
start  of  collaboration,  the  parties  must  resolve  the  relevant  differences. 
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•  Shared  planning.  The  collaborating  parties  have  a  common  set  of  expectations  about 
the  availability  and  applicability  of  methods  to  resolve  the  problem.  A  resolution  to 
the  problem  requires  consensus  of  the  collaborating  parties. 

•  Communications.  A  communications  mechanism  must  exist  to  enable  the 
collaborating  parties  to  exchange  information  about  the  exercise  of  initiative  (“rules 
of  order”),  joint  purpose,  situation  and  candidate  problem  resolutions. 

Studies  in  team  collaborative  decision  making  have  been  conducted  from  different 
disciplinary  viewpoints:  social  and  cognitive  psychologists  (Burke,  2000),  systems 
engineers  (Handley  and  Levis,  2001),  and  organization  theorists  (Marks,  Zaccaro,  and 
Mathieu,  2000).  The  transition  from  traditional  hierarchical  team  problems  to 
distributive  team  decision-making  has  taken  many  modeling  ideologies:  from 
mathematical  optimization  (Miao,  Luh,  and  Kleinman,  1 992),  normative-descriptive 
(Adelman,  Zirk,  Lehner,  Moffett,  and  Hall,  1986)  to  cultural  cognition  studies  (Klein, 
2000). 

A  team  interaction  mental  model  and  situation  awareness  (SA)  have  also  been 
investigated,  and  results  show  that  both  hold  information  concerning  the  roles, 
responsibilities,  communication  patterns,  and  interactions  among  team  members 
(Converse,  Cannon-Boers,  and  Salas,  1991;  Endsley  and  Pearce,  2001).  Testimonies  to 
their  effectiveness  reflect  on  many  axiomatic  definitions. 

Wellens  (1989,  1993)  and  Endley  (2000)  have  distinguished  between  individual 
and  team  SA,  with  team  S  A  referring  to  the  sharing  of  an  SA  regarding  system  events 
(current  and  future  status).  Group  SA  can  be  defined  as  “the  sharing  of  a  common 
perspective  between  two  or  more  individuals  regarding  current  environmental  events, 
their  meaning  and  projected  future  status”  (Wellens,  1989,  p.  6).  Group  SA  and  sharing 
of  information  has  been  shown  to  be  a  critical  factor  in  the  performance  of  airline  crews 
(e.g.,  Foushee  &  Helmreich,  1988).  The  accuracy  of  group  SA  depends  not  only  on  the 
shared  information,  but  also  on  the  shared  “mental  model”  (Rouse,  Cannon-Bowers,  and 
Salas,  1992).  According  to  Wellens  (1993),  the  shared  mental  model  includes:  1)  a 
shared  idea  of  how  the  group  operates  the  system,  and  2)  a  shared  understanding  of  the 
system  problem  the  group  encounters. 
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CHAPTER  4 

SOME  RELEVANT  HUMAN  FACTORS  ISSUES 


4.1  Background 

The  study  of  collaborative  team  decision  making  and  complex  information 
processing  constitutes  a  domain  of  inquiry  for  human  factors  engineers  and  scientists. 

The  complex  nature  of  the  tasks — including  task  difficulties,  task  performance  under 
stress,  latent  issues  of  personality  and  the  nuances  of  collaboration — such  as  conflict 
management,  pose  specific  challenges  that  must  be  resolved  during  the  design  evaluation 
and  prior  to  fielding  work  design  plans  that  involve  human  and  automation.  Many  human 
opportunities  in  the  context  of  this  study  include,  but  are  not  limited  to:  issues  in  working 
memory  (WM),  attention  and  WM  capacity,  controlled  versus  automatic  processing, 
change  blindness,  shared  situation  awareness  (SSA),  and  team  mental  models.  For  this 
study,  we  will  measure  SSA  and  to  some  extent,  use  signal  detection  theory  (SDT)  to 
capture  some  aspects  of  team  vigilance.  A  review  of  some  of  these  human  factors 
concerns  are  summarized  below. 

4.2  Memory  Systems 

The  psychological  mechanisms  that  store  information  relevant  to  the  task  at  hand 
are  referred  to  as  working  memory  (WM).  The  standard  theory  of  WM  includes  separate 
stores  for  verbal  information  and  visual  information  (Baddeley,  1986).  The  earliest 
information-processing  models  posited  one  or  more  sensory  buffers,  a  limited-capacity 
short-term  memory,  and  an  unlimited  long-term  memory.  However,  the  old  notion  of  a 
relatively  passive,  limited-capacity  short-term  memory  has  been  replaced  by  a  more 
active  working-memory  system  that  not  only  holds  and  manipulates  information 
(Daneman  &  Carpenter,  1980),  but  also  attends  selectively  to  one  stimulus  while 
inhibiting  another,  coordinates  performance  on  tasks,  and  switches  strategies  (Baddeley, 
1986).  Oberauer  et  al.  (1996)  summarize  these  processes  as  (1)  simultaneous  storage  and 
processing,  (2)  supervision  or  monitoring,  and  (3)  coordination. 

Fockert,  Ress,  Frith,  &  Lavie  (2001)  tested  the  hypothesis  that  working  memory 
is  crucial  for  reducing  distraction  by  maintaining  the  prioritization  of  relevant 
information.  The  test  used  neuroimaging  and  psychological  experiments  with  humans. 
Participants  performed  a  selective  attention  task  that  required  them  to  ignore  distracter 
faces  while  holding  in  working  memory  a  sequence  of  digits  that  were  in  the  same  order 
(low  memory  load)  or  a  different  order  (high  memory  load)  on  every  trial.  Higher 
memory  load,  associated  with  increased  prefrontal  activity,  resulted  in  greater 
interference  effects  on  behavioral  performance  from  the  distracter  faces,  plus  increased 
face-related  activity  in  the  visual  cortex.  These  findings  confirm  a  major  role  for  working 
memory  in  the  control  of  visual  selective  attention. 

4.3  Attention  and  Working-Memory  Capacity 

Visual  attention  is  particularly  important  to  assess  in  monitoring  tasks.  In  general, 
information  monitoring  consumes  between  20  percent  and  50  percent  of  visual  attention, 
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(Hughes  and  Cole,  1986).  In  complex  task  processing,  it  will  be  of  interest  to  understand 
the  human  ability  to  process  information  in  parallel  or  in  a  time-sharing  manner. 

4.4  Controlled  and  Automatic  Processing 

The  controlled  versus  automatic  processing  distinction  refers  to  the  degree  to 
which  a  knowledgeable  or  skillful  performance  requires  conscious  attentional  resources 
for  successful  operation.  Automatization  occurs  when  cognitive  tasks  are  consistent  in 
their  information-processing  demands  such  that  a  transition  from  controlled  to  automatic 
processing  can  occur  with  practice.  Controlled  processes  are  voluntary,  require  attention, 
and  are  relatively  slow,  whereas  automatic  processes  are  fast  and  do  not  require  attention 
for  their  execution.  Performance  of  novel  tasks  is  typically  thought  to  rely  on  controlled 
processing;  however,  with  extensive  practice,  performance  of  some  tasks  can  become 
automatic  (Posner  &  Snyder,  1975;  Schneider  &  Shiffrin).  Information  processing 
enabled  by  display  automation  in  complex  work  systems  remains  a  fertile  ground  to 
validate  the  performance  of  single  versus  team  operator  performance. 


4.5  Change  Blindness  and  Detection 

In  the  change-detection  procedure,  people  have  to  remember  a  briefly  presented 
visual  display  in  order  to  compare  it  to  a  second  display  and  detect  any  differences 
between  the  two.  Change  blindness  is  the  induced  failure  to  detect  changes  in  a  display 
(Rensink,  O’Reagan,  and  Clark,  1997).  This  blindness  to  change  is  due  to  the  failure  to 
notice  unattended  objects  in  the  direct  line  of  sight.  Change  blindness  typically  occurs 
when  the  visual  scene  is  changed  and  the  eyes  are  in  motion. 

Regardless  of  the  type  of  environment,  it  has  been  shown  that  humans  have 
difficulty  in  detecting  changes  in  scenes  (Rensink,  et  al.,  1997).  An  object  often  changes 
position  within  a  display  or  disappears  completely  and  the  observer’s  eyes  come  to  rest 
without  noticing  the  change  in  magnitude  or  position.  O’Regan  (2000)  provides  an 
overview  of  the  findings  applied  to  the  way  humans  build  internal  representations  with  an 
emphasis  on  the  criticality  of  attention  in  detecting  changes. 

Vogel  et  al.  (2001)  presented  more  evidence  for  a  longer-duration  visual  store.  They 
used  a  change-detection  task  with  displays  consisting  of  colored  squares,  as  illustrated  in 
Figure  l.In  a  trial  in  their  experiments,  a  display  was  presented  for  about  100  ms,  next  the 
screen  remained  blank  for  900  ms,  and  then  a  second  display  appeared  that  was  either 
identical  to  the  first  or  differed  in  the  color  of  one  square  (e.g.,  in  Figure  1,  the  stimulus 
on  the  lower  right  has  changed  color).  Participants  determined  whether  or  not  the  displays 
were  identical.  Vogel  et  al.  showed  that  response  accuracy  declined  as  the  number  of 
squares  in  each  display  increased,  suggesting  that  there  was  some  limited  capacity  system 
for  retaining  the  squares  from  the  first  display.  Further  analysis  suggested  that 
participants  retained  approximately  four  items’  worth  of  information  from  the  first 
display.  Requiring  participants  to  remember  two  digits  while  performing  the  change- 
detection  task  had  no  effect  on  performance,  suggesting  that  storage  was  not  verbal.  Thus, 
there  is  some  evidence  that  unsupported  visual  representations  can  be  retained,  and  that 
retention  is  in  the  form  of  a  visual  code — such  retention  would  be  mediated  by  visual 
working  memory. 
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Figure  1 :  Illustration  of  the  Visual  Change-Detection  Task  used  by  Several 

Researchers  (e.g.,  Vogel,  Woodman,  &  Luck,  2001).  Note:  Stimuli 
are  not  drawn  to  scale. 


4.6  Shared  Situation  Awareness 

Solving  complex  problems  involves  several  cognitive  tasks  that  are  often  studied 
separately.  These  include  situation  awareness  (SA),  decision  making,  and  visual  search. 
Situation  awareness  (Endsley,  1985)  is  the  cognitive  process  of  becoming  aware  of 
numerous  pieces  of  data,  relating  them  to  each  other  and  projecting  their  effect  on  future 
system  states  with  respect  to  the  user’s  goals.  In  crisis  management,  SA  occurs  when 
users  notice  and  respond  to  events  such  as  fires,  emergencies,  and  road  blockages  (due  to 
floods,  lava  flows,  high  winds,  snow,  etc.).  Although  neither  the  concept  nor  the 
measurement  of  SA  has  been  fully  developed,  several  reports  have  explored  the  potential 
and  the  problems  of  the  situation  awareness  concept.  According  to  Endsley  (1985),  SA  is 
the  perception  of  the  elements  in  the  environment  within  a  volume  of  time  and  space,  the 
comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near  future.  This 
definition  lends  itself  to  defining  specific  levels  of  situation  performance: 

•  Level  l  Situation  Awareness:  perception  of  the  status,  attributes,  and  dynamics  of  the 
individual  task-relevant  elements  in  the  environment; 

•  Level  2  Situation  Awareness :  holistic  comprehension  of  the  current  situation,  based 
on  a  synthesis  and  understanding  of  these  elements  in  light  of  one’s  goals;  and 

•  Level  3  Situation  Awareness :  projection  of  the  future  actions  of  these  elements  in  the 
environment,  at  least  in  the  very  near  term. 

The  perception  of  elements  within  the  environment  (Level  1)  is  principally  guided  by  the 
contents  of  working  memory  and  long-term  memory  that  direct  attention,  recognition, 
and  categorization.  Once  perceived,  the  information  is  hypothesized  to  reside  in  working 
memory  where  it  is  combined  with  existing  knowledge  to  develop  a  composite  picture  of 
the  situation. 

4.7  Team  Mental  Models 

Mental  models  are  representations  of  reality  that  people  use  to  understand  specific 
phenomena.  Norman  (in  Gentner  &  Stevens,  1983)  describes  them  as  follows:  "In 
interacting  with  the  environment,  with  others,  and  with  the  artifacts  of  technology,  people 
form  internal,  mental  models  of  themselves  and  of  the  things  with  which  they  are 
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interacting.  These  models  provide  predictive  and  explanatory  power  for  understanding 
the  interaction." 

The  existence  of  mental  models  is  widely  asserted  in  the  literature.  They  are  often 
used  in  a  way  that  is  synonymous  with  "knowledge"  in  general  (Rouse  Cannon-Bowers, 
Salas,  1992).  However,  according  to  Holyoak  (1984,  p.  193),  a  mental  model  is  a 
"psychological  representation  of  the  environment  and  its  expected  behavior."  By 
adopting  a  functional  approach,  Rouse  and  Morris  (1986)  state  that  the  role  of  mental 
models  is  to  provide  a  conceptual  framework  for  describing,  explaining,  and  predicting 
future  system  states. 

4.8  Human  Performance:  Individuals  versus  Teams 

Increasingly,  organizational  research  is  reflecting  the  fact  that  team  strategic 
decisions  are  often  made  by  groups  instead  of  individuals  acting  alone  (e.g.,  Axelrod, 
1976;  Orasanu  &  Salas,  1993).  Thus,  several  constructs  which  have  been  traditionally 
considered  at  the  individual-level  of  analysis  are  now  being  recognized  as  group-level 
phenomena.  A  recent  review  of  group  research  described  the  development  of  "shared 
understanding"  as  an  essential  group  process  (Bettenhausen,  1991,  p.  350). 

It  has  often  been  thought  that  automation  would  reduce  mental  workload,  but  the 
work  of  Edwards  (1976)  and  Weiner  (1988)  have  shown  this  to  be  untrue.  The  work  of 
Sheridan  (1987)  has  shown  that  in  some  cases  the  workload  with  automation  is  greater 
than  with  manual  control.  Traditionally  monitoring  has  been  considered  to  be  a  low 
stimulus  task  leading  to  the  problem  of  ‘vigilance  deficit’  (Duffy  1957).  Signals  are 
usually  clearly  perceivable  when  observers  are  alerted  to  them  but  are  not  compelling 
changes  in  the  observers’  operating  environment.  In  addition,  the  signals  are  usually 
embedded  in  a  context  of  recurrent  non-signal  (neutral)  events,  which,  unlike  signals, 
require  no  overt  response  from  observers.  Idaszak  and  Hulin  (1989)  completed  one  set  of 
experiments  on  the  effects  of  monitoring  automation  with  decision  making  tasks.  They 
used  subjects  to  simulate  process  control  tasks,  one  set  were  active  where  the  subjects 
were  in  control  of  the  process  and  monitored  it  at  the  same  time  and  the  other  where  they 
were  passive  observers  only.  The  active  group  was  faster  at  detecting  out-of-limit 
conditions  and  alarms  than  the  passive  monitors.  In  contrast  Wickens  and  Kessel  (1979) 
found  that  error  detection  was  better  in  automated  rather  than  manual  assignment 
conditions.  Hilbum,  Joma  &  Parasuraman  (1995)  examined  Air  Traffic  Control 
simulation  in  Netherlands  airspace  and  found  benefit  for  automation  on  monitoring 
performance. 
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CHAPTER  5 

DESIGN  OF  EXPERIMENT 


5.1  General  Note 

The  following  experimental  design  components  were  consistent  across  all 
experiments 

•  Criteria  used  to  screen  participants. 

•  Apparatus  used  for  the  experiments. 

•  Design  of  the  experimental  scenarios. 

•  Procedures  and  protocols  for  administering  experiments. 

•  Statistical  Analysis  Software  (SA)  was  used  to  perform  all  analysis.1 

5.2  Participants 

Sixty  students  aged  18-26  years  (mean  age  =  23.7;  standard  deviation  =  3.45) 
participated  fully  in  the  study;  36  participants  were  males  and  24  were  females.  The 
participants  were  undergraduates  and  graduate  students  from  North  Carolina  A&T  State 
University.  To  be  eligible  for  participation,  the  subjects  needed  normal  to  corrected 
normal  vision  and  possess  normal  color  vision.  All  participants  used  in  the  study  met  the 
prescribed  criterion  after  initial  screening  of  a  pool  of  volunteers.  The  60  participants 
were  randomized  into  two  groups  of  three  (G3)  and  five  (G5)  as  follows:  (1)  The  G3 
group  had  3  members  in  a  team  and  participated  in  one  experimental  trials  for  a  total  of 
10  experimental  trials;  (2)  The  G5  group  had  5  members  in  a  team  and  participated  in  one 
experimental  trial  for  a  total  of  6  trials.  The  students  either  did  so  voluntarily  or  were 
compensated  through  a  class  grade  through  agreement  with  a  class  instructor. 

The  sixty  students  used  in  the  experiment  were  determined  by  using  statistical 
power  of  test  analysis.  The  power  of  test  is  the  probability  or  the  ability  of  the  test  to 
detect  an  effect  if  it  exists.  The  estimation  approach  in  conjunction  with  assumed  values 
for  the  expected  standard  deviation  and  margin  of  error  was  used  to  estimate  the  required 
sample  size  for  a  known  power.  The  power  is  indicated  by  (1-P),  where  p  is  the 
probability  of  Type  II  error  (false  hypothesis  is  accepted).  The  significance  level  (1-a) 
where  a  is  the  probability  of  Type  I  error  (true  hypothesis  is  rejected)  typically  varies 
from  0.90  to  0.99.  In  general  power,  the  higher  the  alpha  value,  the  lower  the  power. 

5.3  Apparatus 

A  Gateway  E-Machine  Pentium  IV  computer  with  a  15 -inch  (viewable  space)  flat 
panel  monitor  was  used  to  display  the  experimental  scenarios.  A  computer  program  was 
developed  to  mimic  emergency  incidents  with  a  decision  support  tool  called 


1  Dilorio,  F.C.  (1991).  SAS:  Applications  Programming:  A  Gentle  Introduction.  Belmont,  CA:  Duxbury  Press. 
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MERMAIDS—  Medical  Emergency  Response  using  Military  Asset  in  an  Integrated 
Decision  Support  (MERMAIDS)2.  Figure  2  shows  a  screen  capture  of  the  MERMAIDS 
interface  with  possible  location  of  incidents  and  the  resources  for  emergency  respond. 


Figure  2.  Sample  MERMAIDS  Screen  with  Incident  Map  (right  side)  and  Incident 
Aerial  Photograph  (left  side).  The  red  circle  shows  current  point  of 
incident. 

5.4  Information  Complexity  Test  and  Scenario  Design 

Information  complexity  was  mimicked  with  occurrences  of  emergency  incidents 
by  using  computer  display  signals  to  indicate  the  points  of  incidents.  The  signal  grouping 
varies  from  one  to  four — this  is  known  as  signal  complexity  (SC).  To  add  complexity  to 
the  signals,  timing  was  used  to  control  the  level  of  signal  duration.  The  smaller  the  time 
duration,  the  more  difficult  it  is  to  detect  a  signal  (Sanders  &  McCormick,  1992).  Four 
control  timing  durations  were  utilized:  T1  (0-5)  sec,  T2  (5-10)  sec.,  T3  (10-15)  sec.  and 
Random  (0-15)  sec.  In  the  random  signal  timing,  the  time  was  generated  using  a  uniform 
distribution  with  minimum  value  of  0  and  maximum  value  of  15  seconds.  A  higher 
number  of  SC  with  shorter  timing  defines  the  most  complex  scenario.  The  scenarios 
illustrate  samples  of  signal  monitoring  of  customers  in  a  security  center  in  a  large  urban 
mall,  such  as  the  Mall  of  America  in  Minneapolis.  Equivalently,  it  mimics  the  pilot  who 
must  decide  whether  to  hit  a  hostile  target  or  not — for  example,  Case  3  in  Chapter  1  of 
this  report  in  which  two  American  A- 10  Warthog  pilots  fired  on  what  they  thought  was 
an  Iraqi  armored  column  that  turned  out  to  be  a  group  of  British  fusiliers  and  nine  were 
killed. 

For  a  single  signal  presentation,  it  is  assumed  that  the  tasks  involved  almost 
completely  a  priori  determinable.  It  is  generally  clear  what  information  is  required,  how 


2  Ntuen,  C.A.,  Balogun,  O,  Boyd,  E.,  and  (2005).  Supporting  command  and  control  functions  in  the  emergency  management 
domain  using  cognitive  systems  engineering.  Ergonomics — An  International  Journal  (Accepted:  to  appear). 
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to  find  the  information  and  how  to  assess  relevance.  However,  when  a  “pop-up” 
anywhere  concept  is  embedded  with  signal  timing,  the  issue  of  spatio-temporality 
introduces  a  new  level  of  complexity — making  some  parts  of  the  search  process  or 
information  needed  difficult.  In  our  experiments,  signals  are  presented  to  balance  search 
process  against  prediction  process.  Thus,  it  is  assumed  that  it  is  unclear  what  information 
is  being  sought,  how  to  obtain  relevant  information  and  how  the  searcher  will  know 
where  to  find  relevant  information.  This  is  even  problematic  when  people  are  searching 
for  the  same  pattern  of  information  in  groups. 


5.5  Methodology 


5.5.1  Design  of  Experiment 

The  experiment  was  designed  to  gather  information  on  (a)  Effects  of  team  on 
complex  information  processing  and  (b)  Effects  of  information  complexity  on  team 
detection  sensitivity.  Two  groups  of  participants,  G3  and  G5  discussed  above  were  used. 
Members  were  randomly  assigned  to  a  group  with  no  homogeneity  in  a  group — that  is,  a 
relative  representation  of  genders  as  shown  of  60%  (male)  to  40%  (female).  For  the 
experiment,  issues  of  personality  psychology,  expertise,  and  learning  styles  were  not  of 
interest.  Each  experiment  starts  with  a  random  presentation  of  signals  so  that  the 
complexity  was  balanced.  Each  signal  was  presented  under  four  timing  duration 
controls — Tl,  T2,  T3,  and  Random.  Again,  the  sequence  of  timing  and  signal  complexity 
were  random  to  counter  any  possibility  of  sequence  memorization  by  any  of  the 
participants.  Table  2  shows  the  skeleton  layout  of  the  experimental  design.  Statistically, 
the  design  is  2*4*4  between  groups  and  within  signal  and  timing  design. 


Table  2.  Skeleton  Layout  of  the  Experiment  Design 


G3 

G5 

SC 

Tl 

T2 

T3 
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Tl 

T2 

T3 

Random 
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5.5.2.  Measurement 

Dependent  Variables:  The  dependent  variables  are: 

(a)  Team  SA  (TSA):  This  is  obtained  by  asking  team  members  to  describe,  by 
pressing  the  dedicated  computer  key,  what  they  see  when  a  signal  is  presented. 
The  data  is  summarized  into  percentage  as  a  total  count  of  report  on  “seeing  the 
same  thing.” 

(b)  Team  Informity  (TI):  This  is  obtained  by  tallying  the  observations  on  how  the 

team  members  communicate.  Team  informity  is  defined  by  Hang  (2001)  as  “ . 

the  degree  to  which  the  team  members  access  all  relevant  data,  particularly  target 
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attributes — TI  values  ranges  from  0  (minimum)  to  1  (maximum). The  data  is 
summarized  into  percentage  of  report  on  “Telling  others  what  you  see.” 

(c)  Team  Detection  Sensitivity  (TDS):  This  measures  team  detection  sensitivity  in 
processing  the  displayed  information. 

(d)  Team  Detection  Accuracy  (TDA):  This  measures  the  team  signal  detection 
accuracy.  Data  is  obtained  on  true  detection  rate  (true  positive  detection). 

Independent  variables: 

The  independent  variables  consist  of: 

(a)  the  two  teams  (G3,  G5), 

(b)  four  levels  of  signal  complexity  (1,2,3, 4;  and 

(c)  four  levels  of  timing  control  (Tl,  T2,  T3,  and  Random). 


5.6  Experimental  Procedures 

Participants  were  greeted  and  thanked  for  their  willingness  to  participate  in  the 
experiment.  Participants  were  provided  a  short  briefing  about  the  experiment  including 
requirements,  experimental  objectives,  and  time  requirements.  Those  participants  that 
did  not  meet  the  requirements  or  were  not  willing  to  participate  were  dismissed  from  the 
study  after  the  color  test  and  “20/20”  eye  test.  Participants  were  then  given  a  more 
comprehensive  briefing  of  the  experiment  and  experimental  objectives.  Participants  were 
provided  with  the  required  Internal  Review  Board  (IRB)  content.  Participants  were  asked 
to  sign  a  consent  form  that  indicated  that  they  understood  and  voluntarily  agreed  to  the 
conditions  outlined  in  the  experimental  briefing.  Agreed  participants  were  told  that  they 
may  withdraw  from  the  study  at  any  time. 

Participants  were  told  that  they  will  work  as  a  team  by  monitoring  the 
MERMAIDS  computer  screen  for  incidents  that  will  show  up  as  a  red  circle.  The  incident 
resembles  a  traffic  light  and  will  follow  a  life  cycle  of  “red-green-yellow-red”.  The 
participants  are  to  press  the  signal  number  using  the  computer  mouse  when  the  signal 
turns  red — indicating  a  possible  time  to  respond.  The  color  signals  are  randomly 
generated  using  a  Poisson  distribution  with  mean  of  1200  msec.  Pressing  on  a  yellow 
signal  indicates  false  alarm,  and  pressing  on  a  green  signal  indicates  a  missed 
opportunity.  After  given  general  logistical  instructions  and  time  to  adjust  their 
workstation  for  comfort,  participants  were  given  specific  task  instructions.  The 
participants  were  trained  for  60  minutes  with  sample  trial  scenarios,  and  were  instructed 
to  come  back  after  one  day  for  the  experiment  sessions.  All  group  members  had  to  be 
present  for  the  experiment  to  take  place. 

The  experimenter  was  in  control  of  presenting  all  scenarios  by  running  different 
scenarios  from  the  MERMAIDS  software.  Each  experimental  scenario  took  an  average  of 

48.3  minutes  for  G3  group  with  three  members  and  55.1  minutes  for  G5  group  with  five 
members.  Group  G3  took  a  total  of  7.4  hours  to  complete  all  scenarios  (10  scenarios), 
spanning  a  schedule  of  three  sessions  offered  in  alternate  days.  Group  G5  took  a  total  of 

8.3  hours  to  complete  all  scenarios  (6  scenarios)  spanning  a  schedule  of  four  sessions 
offered  in  alternate  days  to  G3  group.  Three  computer  monitors  were  used  to  display 
information  for  the  G3  group  and  four  were  used  of  the  G5  group.  A  local  area  network 
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(LAN)  protocol  with  common  video  display  card  was  used  to  achieve  these  “common 
picture”  scenarios.  The  set  up  duplicates  a  traffic  monitoring  center  for  a  major  urban  city 
(See  Exhibit  4  below).  All  group  members  have  a  workstation  collocated  to  allow 
discussions  and  information  sharing.  As  the  signal  is  presented,  the  group  members  will 
discuss  among  themselves  and  press  a  dedicated  “F3”  key  on  the  computer  keyboard  to 
indicate  if  they  “see  the  same  incident  signal”.  The  G5  group  will  use  “F5”  for  the  same 
purpose.  A  video  camera  was  used  to  capture  how  the  team  members  exchange 
information  (latter  used  to  define  team  informity). 


Exhibit  4.  Sample  Collaborative  Team  Using  the  Distributed  MERMAIDS 

Information 


5.7  The  Experiment 

The  team  members  are  assigned  their  respective  workstations.  The  experimenter 
starts  the  scenario  by  selecting  appropriate  time  controls  and  signal  levels.  This  is  shown 
in  Figure  3.  After  the  selection,  the  MERMAIDS  system  is  activated  with  the  respective 
signals,  see,  e.g.,  Figure  4. 


Figure  3.  Signal  and  Time  Control  Selection.  Figure  4).  Sample  Screen  with  Three  Signals  with 

Two  Imminent  Threats 

The  participants  used  a  computer  mouse  to  click  on  the  respective  signal,  e.g.,  Figure  5. 
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Figure  5.  Sample  Screen  Where  the  User  Clicks  to  Indicate  Which  Signal  is 
Recognized  in  Multiple  Signal  Situation 


CHAPTER  6 

DATA  COLLECTION  AND  ANALYSIS 


6.1  Data  Collection 


6.1.1  Team  Dimension  Effect 

Data  was  collected  and  summarized  on  team  SA  (TSA)  and  converted  into 
average  percentage  equivalents.  This  is  shown  in  Table  3.  Similarly,  Table  4  shows 
average  team  informity  reflecting  the  frequency  of  information  exchange.  The  data  from 
team  informity  was  obtained  by  reviewing  a  video  that  captures  the  frequency  of 
communication  between  team  members.  Figures  6  and  7  shows  the  graphic  plots  of 
information  on  Tables  3-4. 


Table  3:  Team  SA  (Percentage  of  Members  Reporting  Seeing  the  Same 
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Table  4:  Team  Informity  (Average  Percentage  of  Members  Telling 
_ Others  What  They  See) _ _ _ 
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As  shown  in  Figure  6,  the  distributions  of  mean  percentage  TSA  tend  to  show 
exponentially  decreasing  function  (negative  slopes)  as  the  number  of  signals  to  be 
monitored  increases.  At  signal  control  time  of  10  to  15  sec;  team  G3  shows  the  least 
mean  %  TSA,  while  team  G5  shows  most  reported  TSA.  It  is  not  clear  from  these  results 
whether  teams  with  more  members  tend  to  identify  more  signals  of  interest. 

In  Figure  7,  the  distributions  of  mean  percentage  team  informity  tend  to  show  an 
increasing  exponential  function  (positive  slope)  under  different  signal  time  controls. 
Team  G3  shows  the  least  report  on  information  exchange  among  members  at  signal 
control  time  of  0-5  sec,  but  scored  high  when  the  signal  timing  was  random.  Team  G5 
exhibited  more  information  sharing  among  its  members  at  signal  time  control  of  10-15 
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sec.  The  performance  of  the  teams  at  signal  control  time  of  0-5  sec  confirms  the 
degradation  performance  under  stress  (see,  e.g.,  TADMUS  study  ;  Freeman  &  Wolf, 
1996).  The  increase  in  information  sharing  after  latency  (i.e.,  after  individual  or  group 
search  for  signal)  may  be  responsible  for  the  average  increase  on  informity  metric 
between  10-15  sec  signal  timing  controls. 
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Figure  6.  A  Plot  of  Team  SA  Report  by  Teams 
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Figure  7.  A  Plot  of  Team  Informity  Reported  by  Teams 


6.1.2  Detection  Accuracy 

Data  on  signal  detection  accuracy  was  collected  for  all  conditions  and  the  two  teams. 
Table  5  shows  the  data  for  groups  G3  and  G5.  The  corresponding  bar  charts  are  shown  in 
Figures  8-9,  respectively. 


Table  5:  Average  Signal  Detection  Accuracy  (%) 


(0-5)sec 

(5- 

10)sec 

(10-15)se 

c 

Random  i 

o 

CD 

LO 

1 

o 

Signal 

complexity 

Team=3 

Team=5 

Team=3 

Team=5 

Team=3 

Team=5 

Team=3 

Team=5 

1 

90.33 

91.7 

92.67 

93.6 

91.3 

98.4 

88.9 

85.6 

2 

80.65 

84.1 

90.16 

89.5 

95.6 

94.4 

81.5 

86.1 

3 

79.7 

80.55 

88.24 

90.8 

94.3 

91.7 

79.67 

80.66 

4 

77.5 

80.2 

83.6 

88.4 

85.3 

93.22 

74.3 

70.3 
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Figure  8:  Frequency  Plot  of  %  Detection  Accuracy  for  a  Team  of  3  Members  (G3) 


Figure  9:  Frequency  Plot  of  %  Detection  Accuracy  for  a  Team  of  5  Members  (G5) 


6.1.3  Detection  Response  Time 

Data  on  signal  detection  time  (in  msec)  was  collected  for  all  conditions  and  the  two 
teams.  Table  6  shows  the  data  for  groups  G3  and  G5.  The  corresponding  bar  charts  are 
shown  in  Figures  10-11,  respectively 


Table  6:  Average  Signal  Detection  Time  (msec) 
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717 
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Figure  10:  Signal  Detection  Time  by  G3  Team 


6.1.4  Signal  Detection  Sensitivity  Analysis 

Signal  detection  theory  (SDT)  provides  another  tool  for  analyzing  signal  detection  data. 
The  SDT  allows  for  the  quantification  of  human  tendencies  to  make  correct  judgments, 
misses,  and  false  alarms.  Such  quantification  serves  as  an  analytical  metric  to  measure 
human  performance  in  signal  detection  tasks  (Wickens,  1984).  Signal  detection  theory 
was  used  to  measure  the  participant’s  ability  to  discriminate  incidents  considered  to  be  a 
threat  by  judging  whether  the  signal  is  present  or  absent.  The  SDT  is  a  function  of  two 
processes:  the  observer’s  perceptual  sensitivity  and  response  bias.  This  theory  enables 
separation  of  the  effects  of  perceptual  sensitivity,  in  this  case,  the  observer’s  ability  to 
discriminate  signal  occurrence  from  response  bias.  Response  bias  represents  the 
observer’s  willingness  to  say  "yes"  or  "no." 

In  a  signal  detection  experiment,  events  can  be  categorized  in  a  2  x  2  matrix 
showing  the  event  that  occurred  and  the  observer’s  response  to  that  event.  Figure  7  shows 
the  four  possible  categories.  A  hit  is  said  to  occur  when  a  signal  is  present  and  the 
observer  reports  that  the  signal  is  present.  A  false  alarm  occurs  when  a  signal  is  absent 
and  the  observer  reports  that  the  signal  is  present.  A  miss  occurs  when  a  signal  is  present 
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and  the  observer  reports  that  the  signal  is  absent.  A  correct  rejection  occurs  when  a  signal 
is  absent  and  the  observer  reports  that  the  signal  is  absent.  Table  7  illustrates  this 
situation. 

Data  on  signal  detection  frequency  was  collected  for  all  conditions  and  the  two 
teams.  Tables  8-9  show  the  data  for  groups  G3  and  G5.  Figures  12-  16  are  used  to  capture 
the  Relative  Operating  Characteristics  (ROC)  based  on  the  Equation  1  (Swets,  1996): 

h  =  p(l-p)f  (1) 

where  h  is  the  probability  of  a  “hit”;  f  is  the  probability  of  “false  alarm”,  and  p  is  the  “hit” 
probability  corrected  chance  success,  and  is  often  defined  by  p  =  (h-f)/(l-f).  In  Tables  8- 
9,  true  positive  is  denoted  by  “TP”,  and  false  positive  by  “FP”.  Both  FN  (false  negative) 
and  TN  (true  negative)  are  redundant  information  for  signal  detection  analysis.  The  ROC 
plots  for  different  experimental  scenarios  are  shown  in  Figures  12-16. 

Table  7.  An  Illustration  of  Signal  Detection  Data  Classification 


!... .  j 

Participant’s  response 

Yes  No 

Signal  Present 

TP 

FN 

Signal  Absent 

FA  TN 

In  Figures  12-16,  the  lower  parts  of  the  ROC  indicate  less  sensitivity  of  the 
observer  to  signal.  As  will  be  described  in  the  next  section,  the  metric,  d’  (pronounced  D 
prime)  is  used  to  detect  the  levels  of  the  sensitivity  for  specific  signal  strengths. 


Table  8:  Hit  Rate  (True  Positive)  and  False  Alarm  (False  Positive)  Counts  for  3  Team 
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Figure  13(a):  ROC  for  (5-10  sec);G3  Figure  13(b):  ROC  for  (5-10  sec);G5 


Figure  15(a):  ROC  for  (5-10  sec);G3 


Figure  15(b):  ROC  for  (5-10  sec);G5 


6.1.5  Signal  Detection  Sensitivity  (d’) 

Sensitivity  is  measured  typically  by  an  index  called  d'.  The  sensitivity,  d'  is  a 
statistic  that  measures  the  difference  between  “hit”  rate  and  “false  alarm  ”  rate.  Flowever, 
d'  is  not  simply  a  geometric  distance;  rather,  it  is  the  difference  between  the  z-transforms 
(assumes  a  normal  distribution  with  mean  zero  and  standard  deviation,  1)  of  these  2  rates 
(Green  and  Swets,  1988)  and  is  denoted  by  Equation  2  as: 

d'  =  z(H)  -  z(F)  (2) 

Tables  10  and  1 1  shows  the  calculated  d'  for  teams  G3  and  G5,  respectively.  The 
higher  values  of  d'  measures  how  good  the  signal  is  detected  by  the  group.  For  example, 
in  team  G3,  single  or  multiple  signals  presented  randomly  (random  time  control)  tends  to 
reduce  the  operator  sensitivity.  It  is  not  clear  why  three  signals  with  timing  control  levels 
of  5-10  sec  and  10-15  sec  induce  poor  sensitivity.  The  best  d'  for  G3  team  occurs  with 
monitoring  4  signals  at  10-15  sec  time  control  (d'  =  5.15)  and  also  when  2  signals  are 
presented  at  the  same  time  control.  With  team  G5,  higher  d'  occurs  at  timing  control  of  5- 
lOsec.  for  a  single  signal  monitoring  task  (d'  =  2.96);  the  worst  d'  occurs  when  2  signals 
are  monitored  under  random  time  control. 


Table  10:  Detection  Sensil 

tivity  (cf)  Values  for  Group  with  3  Team  Members  (G3) 

Time 
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Signal  Complexity 
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Random 
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0.7 

0.63 
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2 

0.64 
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0 

3 

0.26 

0 
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4 

0.78 
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Tablel  1:  Detection  Sensitivity  (d1)  Values  for  Group  with  5  Team  Members  (G5) 
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Figure  16(a):  d'  for  a  Team  of  3  Members  (G3) 


Figure  16(b):  d'  for  a  Team  of  5  Members  (G5) 


CHAPTER  7 
PERMANCE  ANALYSIS 


7.1  Background 

In  this  study,  the  main  performance  analyses  are  associated  with  two  major  tasks: 

Task  1:  An  Assessment  of  Collaborative  Team  Characteristics  Using  Simulated 
Emergency  Response  Teams 

One  of  the  fundamental  challenges  confronting  team  performance  is  understanding 
the  characteristics  of  the  team.  A  team  characteristic  is  multidimensional  and  requires  a 
multidisciplinary  approach.  For  this  reason,  different  disciplines  have  different 
descriptions  of  what  a  team  is  which,  at  times,  conflict  with  our  understanding  of 
homogenous  group  characteristics.  In  this  task,  two  main  hypotheses  are  investigated: 

(1)  There  is  a  difference  in  mean  team  SA(how  teams  see  the  same  thing)  between  a 
team  with  3  members  and  teams  with  5  members  when  performing  signal 
monitoring  tasks. 

(2)  There  is  a  difference  in  mean  team  informity  (how  teams  exchange  information 
on  what  they  see)  between  a  team  with  3  members  and  a  team  with  5  members 
when  performing  signal  monitoring  task 

Task  2:  Assess  the  Dimensions  of  Team  Performance  with  Respect  to  Information 
Processing 

In  this  task,  we  measure  objective  attributes  of  performance  in  terms  of  detection 
accuracy  and  detection  time.  Thus,  two  hypotheses  were  tested: 

(1)  There  is  a  difference  in  mean  signal  detection  accuracy  between  a  team  with  3 
members  and  a  team  with  5  members  when  performing  signal  monitoring  tasks. 

(2)  There  is  a  difference  in  mean  signal  detection  time  between  a  team  with  3 
members  and  a  team  with  5  members  when  performing  signal  monitoring  tasks 

In  testing  the  four  hypotheses,  possibilities  of  interaction  effects  were  analyzed.  All  tests 
were  conducted  using  a  level  of  significant  (  a)  of  0.05. 


7.2  Analysis  of  Variance  for  Statistical  Measures 


An  ANOVA  was  conducted  to  investigate  mean  differences  in  each  of  the 
performance  measures  and  to  validate  the  hypotheses  of  the  study.  There  are  three  levels 
of  main  effects:  two  levels  of  teams,  denoted  by  A  (G3,  G5),  four  levels  of  signal 
complexity,  denoted  by  B  (1,2, 3, 4),  and  four  levels  of  time  control,  denoted  by  C  (0-5sec, 
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5-10sec,  10-15sec,  Random),  respectively.  The  experiment  is  2*4*4  between  teams  and 
within  signal  complexity  and  timing.  All  experimental  blocks  had  1 0  replications,  leading 
to  320  observations. 


7.2.1  The  Difference  in  Mean  Team  SA.  (How  teams  see  the  same  thing)  between  a 
team  with  with  3  members  and  a  team  with  5  members  when  performing  signal 
monitoring  tasks. 

The  corollary  of  hypothesis  in  7.2.1  is  that  there  is  an  interaction  effect  between  teams 
and  signal  complexity  and  /or  signal  time  control.  Table  12  gives  the  ANOVA  results. 


Table  12:  Three-Way  ANOVA  Using  Mean  Team  SA  Data  (Seeing  the  Same  Thing) 


Source 

DF 

SS 

MS 

F 

P 

Decision 

Team  (A) 

1 

27.6534 

27.6534 

5.69 

0.037 

** 

Signal 

3 

47.443 

15.814 

3.524 

0.0001 

** 

Complexity  (B) 

Time  Control 

3 

104.495 

34.832 

7.167 

0.0411 

** 

(C) 

AB 

3 

42.982 

14.327 

2.948 

0.168 

* 

AC 

3 

77.7114 

25.904 

5.333 

0.992 

* 

BC 

9 

78.557 

8.729 

1.796 

0.0026 

& 

ABC 

9 

161.4006 

17.933 

3.69 

0.321 

* 

Error 

288 

1399.68 

4.86 

Total 

319 

1939.922 

Reject  null  hypothesis  (not  statistically  significant  based  on  F-statistics  at  0.05). 
*  Test  is  not  statistically  significant.  &  Accept  null  hypothesis. 


As  shown  in  Table  12,  the  equality  of  Team  SA  between  teams  was  rejected  (Fo.os 
=  3  <  Fi,  288  (5-69);  p=  0.037).  Similarly,  signal  complexity  was  not  significant  (Fo.o5= 

2.26  <  F3,288  (3.524);  p  =  0.0001);  Time  control  was  not  significant  (Fo.os  =  2.6  <  F3>288 
(7.167);  p  =  0.041 1).  An  interaction  effect  was  observed  between  signal  complexity  and 
time  control  (Fo.os  =  1.88  >  F^  288  (1.796);  p  =  0.0026).  Thus,  we  accept  the  hypothesis 
that  signal  complexity  and  signal  timing  control  may  interact  to  affect  signal  monitoring 
performance.  Figures  17(a)-  17(b)  are  used  to  show  the  interaction  points.  In  both  figures, 
interactions  are  more  pronounced  at  a  signal  timing  control  of  10-15  sec. 
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Figure  17(a).  Time-Signal  Complexity  Figure  17(b)  Time-Signal  Complexity  for  G5 

Interaction  for  G3 


7.2.2  Difference  in  Mean  Team  Informity  (how  teams  share  information  on  what 
they  see)  between  a  team  with  3  members  and  a  team  with  5  members  when  performing 
signal  monitoring  tasks. 

The  corollary  of  hypothesis  in  7.2.2  is  that  there  is  interaction  effect  between  teams  and 
signal  complexity  and  /or  signal  time  control.  Table  13  gives  the  ANOVA  results. 


Table  13:Three-Way  ANOVA  Using  Mean  Team  Informity  Data  (Telling  What  You  See): 


Source 

DF 

SS 

MS 

F 

P 

Decision 

Team  (A) 

1 

19.532 

19.532 

3.82 

0.238 

* 

Signal 

3 

21.321 

7.7107 

1.39 

0.001 

& 

Complexity  (B) 

Time  Control 

3 

41.8755 

13.9585 

2.73 

0.167 

* 

(C) 

AB 

3 

79.456 

26.485 

5.18 

0.0001 

** 

AC 

3 

45.097 

15.03 

2.94 

0.316 

* 

BC 

9 

56.739 

6.304 

1.233 

0.093 

* 

ABC 

9 

155.078 

17.23 

3.37 

0.077 

* 

Error 

288 

1472.544 

5.113 

Total 

319 

1891.6385 

Reject  null  hypothesis  (not  statistically  significant  based  on  F-statistics  at  0.05). 
*  Test  is  not  statistically  significant.  &  Accept  null  hypothesis. 


As  shown  in  Table  13,  the  equality  of  team  informity  between  teams  was  rejected 
(Fo.os  =  3  <  Fi, 288  (3.82);  p=  0.0.238).  Signal  control  was  not  significant  (Fo.os  =  2.6  <  F3; 
288  (2.73);  p  =  0.167).  There  were  no  interaction  effects.  Results  proved  the  null 
hypothesis  for  signal  complexity  to  be  accepted  (Fo.os  =  3  >  F3, 288  (1  -39);  p  =  0.001). 
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7.2.3  There  is  a  difference  in  mean  signal  detection  accuracy  between  a  team  with 
3  members  and  a  team  with  5  members  when  performing  signal  monitoring 
tasks. 

The  corollary  of  hypothesis  in  7.2.3  is  that  there  is  an  interaction  effect  between  teams 
and  signal  complexity  and  /or  signal  time  control.  Table  14  gives  the  ANOVA  results. 


Table  14:  Three-Way  ANOVA  Using  Mean  Detection  Accuracy  Data 


Source 

DF 

SS 

MS 

F 

P 

Decision 

Team  (A) 

1 

9.2565 

9.2565 

2.493 

0.631 

* 

Signal 

3 

18.6467 

6.2156 

1.674 

0.00001 

& 

Complexity  (B) 

Time  Control 

3 

10.459 

3.4865 

0.939 

0.007 

& 

(C) 

AB 

3 

29.7077 

9.903 

2.667 

0.364 

* 

AC 

3 

20.4178 

6.806 

1.833 

0.157 

* 

BC 

9 

25.6643 

2.852 

0.768 

0.0003 

& 

ABC 

9 

85.581 

9.509 

2.561 

0.081 

* 

Error 

288 

1069.344 

3.713 

Total 

319 

1269.077 

Reject  null  hypothesis  (not  statistically  significant  based  on  F-statistics  at  0.05). 
*  Test  is  not  statistically  significant.  &  Accept  null  hypothesis. 


As  shown  in  Table  14,  the  equality  of  mean  detection  accuracy  between  teams 
was  rejected  (Fo.os  =  3  <  Fi(288  (2.493);  p-  0.631).  Similarly,  interactions  between  teams 
and  signal  complexity  and  teams  with  signal  time  control  were  not  observed.  Signal 
complexity  was  significant  (Fo.os  -  3  >  F3, 288  (1 .674);  p  =  0.00001).  Signal  time  control 
was  significant  (Fo.os  =  3  >  F3;  288  (0.939);  p  =  0.0.007).  There  was  interaction  between 
signal  complexity  and  signal  time  control  (Fo.os  =  1.88  >  Fg;288  (0.768);  p  =  0.0003).  Thus, 
we  accept  the  hypothesis  that  signal  complexity  and  signal  timing  control  may  interact  to 
affect  signal  monitoring  performance.  Figures  18(a)-  8(b)  are  used  to  show  the  interaction 
points.  In  both  figures,  interactions  are  more  pronounced  at  signal  timing  control  of  10-15 
sec;  Team  G5  show  two  possible  interactions  at  Random  signal  timing  for  signal 
complexities  1  and  2,  respectively. 


Figure  18(a):  Mean  Detection  Accuracy  Interaction  for  G3 


Figure  18(b):  Mean  Detection  Accuracy  Interaction  for 
G5 
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7.2.4  There  is  a  difference  in  mean  signal  detection  time  between  a  team  with  3 
members  and  a  team  with  5  members  when  performing  signal  monitoring 
tasks. 

The  corollary  of  hypothesis  in  7.2.4  is  that  there  is  interaction  effect  between  teams  and 
signal  complexity  and  /or  signal  time  control.  Table  14  gives  the  ANOVA  results 

As  shown  in  Table  15,  the  equality  of  mean  detection  time  between  teams  is 
accepted  (Fo.os  =  3  <  Fi;  288  (1-773);  p=  0.0001).  Similarly,  signal  complexity  has  effect  on 
mean  detection  time  (Fo.os  =  3  >  Fi,  288  (0.9  1  5);  p  =  0.036);  Signal  time  control  was 
significant  (Fo.os  =  3  >  F3i288  (2.38);  p  =  0.0.00001);  There  were  interaction  between 
teams  and  signal  time  control  (Fo.os  =  1.88  <  F9j  288  (1.291);  p=  0.001);  Signal  complexity 
and  signal  time  control  (Fo.os  =  1 .88  >  F9j  288  (0.77);  p  =  0.00001).  Thus,  we  accept  the 
hypothesis  that  teams  have  effect  on  signal  performance.  The  statistical  tables  in  Chapter 
6  show  the  mean  detection  times.  Signal  complexity  and  signal  timing  control  may 
interact  to  affect  signal  monitoring  performance.  Figurel9  shows  the  interaction  between 
teams;  Figure  20(a)  and  20(b)  shows  the  interaction  between  signal  complexities  and 
signal  timing  control. 


Table  15:  Three-Way  ANOVA  Using  Mean  Detection  Time  Data  :  10  Replications  per 
Experiment  Block. 


Source 

DF 

SS 

MS 

F 

P 

Decision 

Team  (A) 

1 

5.1116 

5.1116 

1.773 

0.0001 

& 

Signal 

3 

7.9138 

2.638 

0.915 

0.036 

& 

Complexity  (B) 

Time  Control 

3 

20.58462 

6.862 

2.38 

0.00001 

& 

(C) 

AB 

3 

30.7558 

10.252 

3.556 

0.064 

* 

AC 

3 

11.16586 

3.7219 

1.291 

0.001 

& 

BC 

9 

19.97919 

2.2199 

0.77 

0.00001 

& 

ABC 

9 

65.72375 

7.3026 

2.533 

0.083 

* 

Error 

288 

830.304 

2.883 

Total 

319 

991.53862 

Reject  null  hypothesis  (not  statistically  significant  based  on  F-statistics  at  0.05). 
Test  is  not  statistically  significant.  &  Accept  null  hypothesis. 
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Figure  19:  Graphical  Display  of  Team  and  Signal  Control  Interaction  for  Detection  Time 
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Mean  Detection  Time  (msec) 


(5-10)sec  (10-15)sec  Random 

Signal  Control  Time  (sec) 


Figure  20(a):  Mean  Detection  Time  Interaction  for  G3  Figure  20(b):  Mean  Detection  Time  Interaction  for  G5 
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CHAPTER  8 

DISCUSSIONS  AND  CONCLSIONS 


8.1  Discussions 

Most  tasks  today  depend  on  information  displays,  and  more  so,  by  teams  of 
people  using  the  displayed  information  for  key  decisions.  Significant  uses  of  information 
displays  and  visualization  by  teams  span  across  important  task  domains  of  national 
security  significance:  monitoring  city  traffic  for  unusual  events  by  a  team  of  controllers,  a 
team  of  monitoring  traffic  and  other  events  in  a  mall  using  cameras,  a  pilot  and  co-pilots 
monitoring  the  cockpit  for  flight  information,  or  monitoring  a  nuclear  facility  by  a  team 
of  operators.  Team  characteristics,  coupled  with  information  characteristics  can  lead  to 
degradation  in  performance  rather  than  improve  it — especially  as  information  becomes 
more  dense  and  complex — for  example,  a  team  of  two  operators  watching  traffic  in 
Minneapolis  mall  with  over  500  cameras. 

This  research  was  conducted  to  understand  this  phenomenon — whether  complex 
information  processing  affects  teams  performance — specifically,  using  information 
display  and  visualization.  The  study  used  both  subjective  and  objective  measures.  The 
subjective  measures  were  used  to  assess  team  effect  in  terms  of  team  SA  and  team 
information  sharing.  The  objective  measures — signal  sensitivity  (d'),  time  and  accuracy 
metrics  were  used  to  analyze  effects  of  task  performance,  specifically,  signal  monitoring. 
The  following  results  were  observed: 

(1)  There  was  no  difference  between  the  teams  used  in  the  study  with  respect  to  how 
they  reported  seeing  the  same  thing  (team  SA)  and  exchanging  information  (team 
informity). 

(2)  For  the  team  with  3  members,  the  distributions  of  the  mean  percentages  of  same 
team  SA  tend  to  follow  an  exponentially  decreasing  function  (negative  slopes)  as 
the  number  of  signal  presented  increases.  The  team  with  5  members  reported 
more  team  SA.  It  is  not  clear  from  these  results  whether  teams  with  more 
members  tend  to  identify  more  signals  than  teams  with  less  number  of  members 
(G3). 

(3)  It  is  possible  that  signal  complexity  and  signal  timing  control  may  interact  to 
affect  signal  monitoring  performance — which  in  turn  affects  the  way  people 
report  what  they  see  under  stress.  This  was  apparent  by  the  interaction  between 
signal  complexity  and  signal  timing  control. 

(4)  The  distributions  of  mean  team  informity  show  increasing  positive  slopes  under 
different  signal  timing  controls.  Team  G3  shows  the  least  report  on  information 
exchange  among  members  at  signal  control  time  of  0-5  sec,  but  scored  high  when 
the  signal  timing  was  random.  Team  G5  exhibited  more  information  sharing 
among  its  members  at  signal  time  control  of  10-15  sec.  The  performance  of  the 
teams  at  signal  control  time  of  0-5  sec  confirms  the  degradation  performance 
under  stress  (see,  e.g.,  TADMUS  study).  The  increase  in  information  sharing  after 
latency  (i.e.,  after  individual  or  group  search  for  signal)  maybe  responsible  for 
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the  average  increase  on  informity  metric  between  10-15  sec  signal  timing 
controls. 

(5)  Signal  detection  sensitivity  was  calculated  across  all  experimental  blocks.  The 
results  showed  some  results  that  need  further  investigation.  For  example,  in  team 
G3,  single  or  multiple  signals  presented  randomly  (random  time  control)  tends  to 
reduce  the  operator  sensitivity.  It  is  not  clear  why  three  signals  with  timing 
control  levels  of  5-10  sec  and  10-15  sec  induce  poor  sensitivity.  The  best  d'  for 
G3  team  occurs  with  monitoring  4  signals  at  10-15  sec  time  control  (d'  =  5.15) 
and  also  when  2  signals  are  presented  at  the  same  time  control.  With  G5  team,  a 
higher  d'  occurs  at  timing  control  of  5-10sec.  for  a  single  signal  monitoring  task 
(d’  =  2.96);  the  worst  d'  occurs  when  2  signals  are  monitored  under  random  time 
control. 

(6)  Team  composition  has  no  effect  on  detection  accuracy.  However,  the  signal 
timing  and  complexity  showed  some  statistical  significance  indicating  that 
information  complexity  affects  performance  in  terms  of  detection  accuracy.  The 
histograms  plotted  in  Chapter  6  show  some  points  where  such  differences  occur. 
Detection  accuracy  differences  can  be  caught  by  analyzing  the  differences  in 
mean  performance  at  the  points  of  interaction.  In  this  study,  interactions  are  more 
pronounced  at  signal  timing  control  of  10-15  sec.  G5  team  shows  two  possible 
interactions  at  Random  signal  timing  for  signal  complexities  1  and  2,  respectively. 

(7)  In  terms  of  detection  (response)  time,  both  teams  tend  to  perform  almost  equally. 
This  result  may  be  different  when  comparing  experts  and  novices.  Note  that  teams 
were  almost  a  balance  of  males  and  females  and  no  classification  was  made  as 
how  the  team  was  to  perform  the  signal  detection  tasks. 

(8)  Both  signal  complexity  and  signal  control  timing  showed  effects.  There  were 
interactions  between  teams  and  signal  time  control — indicating  the  possibility  of 
effect  of  constraints  on  timing.  For  example,  with  a  short  timing  of  0-5  sec, 
performance  is  less  due  to  shortness  of  the  signal  duration.  However,  as  the  signal 
duration  increases  to  10- 15  sec,  people  can  take  more  time  to  scan  the 
environment  for  the  target  signal. 


8.2  Conclusions 

The  results  obtained  in  this  study  indicate  that  non-classified  team  composition 
(with  assumption  of  homogeneity  in  task  performance)  does  not  affect  signal  monitoring 
task  performance.  However,  caution  must  be  exercised  in  this  observation  since  detection 
sensitivity  (d')  may  produce  different  results  by  teams  as  shown  in  the  study.  It  is  also 
observed  as  an  affirmation  in  past  studies  (Vogel,  et  al.,  2001)  that  information 
complexity — a  function  of  number  of  signals  monitored  and  signal  control  time  for  signal 
shell  life  affects  performance — either  in  detection  accuracy  or  detection  (response)  time. 

The  study  has  many  implications  for  a  complex  information  processing 
environment.  First,  understanding  team  SA  and  how  teams  exchange  information  in 
either  a  confined  environment,  such  as  a  pilot  or  a  ship  sonar  operator,  or  in  teams  who 
may  be  dispersed  geographically,  but  use  the  same  display  and  visualization  tools. 
Second,  signal  vigilance  research  (Carter,  1982)  has  traditionally  observed  the 
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performance  of  one  operator.  This  study  will  allow  for  signal  vigilance  research  to  be 
extended  to  team  situations  where  team  performance  in  automation  monitoring — such  as 
multiple  operators  monitoring  a  missile  control  station,  nuclear  power  plant,  or  a  port 
security  monitoring  information  for  a  potential  terrorist,  can  be  assessed.  Third,  it  will  be 
advantageous  to  extend  this  experiment  to  more  realistic  domains.  For  example,  at 
present,  it  is  not  known  how  the  installation  of  multiple  cameras  in  high-risk  monitoring 
environments  affects  the  operator’s  performance-for  example,  monitoring  a  stadium 
during  a  Super  Bowl  game.  Workload  measures — both  subjective  and  objective  can  be 
used  to  develop  performance  metrics  for  complex  information  processing  environment. 
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